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Tide: AUTOMATED CYTOLOGICAL SPECIMEN 
CLASSIFICATION SYSTEM AND METHOD 



This application is a continuation-in-part of U.S. patent application Serial No. 
179,060, filed April 8, 1988, and/or U.S. patent application Serial Nos. 420,105 and 
425,665, filed October 11, 1989, and October 23, 1989, respectively, both entitled 
Automated Cytological Specimen Classification System And Method, the entire 
disclosures of which are hereby incorporated by reference. 



ICAL FIELD 

This invention relates generally, as indicated, to cell classification and, more 
particularly, to a system for increasing the speed and accuracy of cervical smear 
analysis. 



BACKGROUND OP THE TNVPMTTOM 

The examination of a cervical smear by what often is referred to as a Pap test 
is a mass screening cytological examination which currently requires individual visual 
inspection by a person of virtually all of the approximately 100,000 cells on a typical 
slide. The te$t, therefore, suffers from a high false negative rate due to the tedium 
and fatigue associated with this requirement for exhaustive search. 

Prompted by the clear commercial potential for automated cervical smear 
analysis, several attempts to this end have been made heretofore. These attempts 
have proven to be unsuccessful at least partly because they could not accommodate 
overlapping cells as are typically found in the Pap smear. To circumvent the 
classification problems created by overlapping cells, specialized "monolayer 
preparations" have been prepared. A monolayer preparation is a specially prepared 
smear in which the cervical cells are centrifuged and filtered so that only a single 
layer of cells results. Besides serious cell preservation and cell transportation 
problems, the expense and time involved in the monolayer preparation precludes its 
use as a population screening substitute for the Pap smear. 

Even when limited to the non-overlapping cell images provided by the 
monolayer preparation, prior art attempts at automated cytological classification have 
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not been able to process cervical smear images at anything close to manual processing 
time. Many of these attempts at automated cytological classification have relied on 
feature extraction algorithms which attempt to select and to measure some feature 
within the image, e.g. , the shape of the cell nucleus. Feature extraction algorithms 
5 have failed due to the inability to segment the image into the components which 

require measurement. One cannot measure nuclear size, for example, unless the 
image is segmented so that the cellular nuclei are identified. Template matching, in 
which an actual image (not a mathematical quantity) is compared with stored 
exemplar images also has not been successful since it is computationally intensive and 
10 the infinite variety of possible Pap smear images or scenes would require an excessive 

number of exemplar image comparisons. The distinction between feature extraction 
and template matching is outlined in the Collings reference on pages 1 through 5 
while image segmentation techniques are discussed in Chapter 7 of the Gonzalez 
reference. 

15 example of the limitations of the prior art can be found in the 1987 

reference entitled "Automated Cervical Screen Classification- by Tien et al, identified 
further below. 

Background references of interest are, as follows: 
Rumelhart, David E. and McClelland, James L., "Parallel Distributed 
20 Processing," MTT Press, 1986, Volume 1; 

Tien, D. et al, "Automated Cervical Smear Classification," Proceedings of the 
IEEE/Ninth Annual Conference of the Engineering in Medicine and Biology Society, 
1987, p. 1457-1458; 

Hecht-Nielsen, Robert, "Neurocomputing: Picking the Human Brain," tf.br 
25 Spectrum, March, 1988, p. 3641; and 

Lippmann, Richard P., "An Introduction to Computing with Neural Nets," 
IEEE ASSP Magazine, April, 1987, p. 4-22. 



30 



BRIEF SUMMARY OP THP TNVENTTOM 

An object of the invention is to automate at least part of the classification 
procedure for cytological specimens. 
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Another consistent objective is to provide semi-automation in a cytological 
specimen classification apparatus and method, whereby at least part of the cell 
classification procedure may be carried out by a human being. 

Consistent with the foregoing, an object of the present invention is to classify 
cytological specimens into categories, for example, categories of diagnostic 
significance, and, more particularly, to automate at least a part of such classification 
procedure. 

As used herein the term "automated" means that at least part of the apparatus 
is automated; in the preferred embodiment a portion of the method is carried out by 
a person. 

Briefly, according to one embodiment, the invention includes an initial 
classifier (sometimes referred to as a primary classifier) preliminarily to classify a 
cytological specimen and a subsequent classifier (sometimes referred to as a 
secondary classifier) to classify those portions of the cytological specimen selected by 
the initial classifier for subsequent classification. 

According to one embodiment, the invention includes an initial classifier 
(sometimes referred to as a primary classifier) preliminarily to classify a cytological 
specimen, a subsequent classifier (sometimes referred to as a secondary classifier) to 
classify those portions of the cytological specimen selected by the initial classifier for 
subsequent classification, and a tertiary classification to determine characteristics of 
or to classify those portions of the cytological specimen that are selected by the 
subsequent classifier for further classification. 

In one embodiment the primary classifier performs a low level morphological 
feature screening function on the entire image while the secondary classifier performs 
a high level pattern matching identification on those images not screened out by the 
primary classifier. 

In an embodiment of the invention the primary classifier classifies specimens 
according to size criteria and integrated optical density. 

In an embodiment the secondary classifier is a neural net. 
In an embodiment the tertiary classifier may be a person. 
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Li an embodiment the present invention performs its classification of a group 
of specimens within the period of time typically consumed for this task by careful 
manual screening (i.e., approximately six minutes/specimen) or fester. 

In an embodiment the present invention performs its classification on 
5 cytological specimens which contain the numbers and types of objects other than 

single layers of cells of interest that are typically found in cervical Pap smears (e.g., 
clumps of cells, overlapping cells, debris, clumps of leucocytes, bacteria, mucus). 

In an embodiment the present invention performs the above- described 
classification on cervical smears for the detection of pre-malignant and malignant 
10 cells. 

In an embodiment the present invention displays, e.g., on a monitor or other 
display medium, cells adjacent or near one or more exemplary cells having features 
distinctive of a certain cell classification, such as large dark nuclei for malignant or 
pre- malignant cells, to facilitate, by comparison, cell screening by a person. 
15 111311 embodiment the present invention performs its classification with smaller 

felse negative error rates than those typically found in conventional manual cervical 
smear screening. 

In an embodiment of the present invention classification of cytological 
specimens into medically significant diagnostic categories will be more reliable, i.e., 
20 will have lower false negative error rates, than present methods. 

In an embodiment the cytological classification system of the present invention 
does not require a modification in the procedure by which cellular specimens are 
obtained from the patient, i.e., standard Pap smears are used for its input 

In an embodiment the cytological classification system of the present invention 
will permit reliable classification within processing time constraints that permit 
economically viable operation. 

In an embodiment of the invention classification of a cytological specimen is 
made by a person, and subsequent automated (or semi-automated) classification of 
selected specimens, such as those primarily noted as negative by such person, or, if 
desired, of all specimens, then may be carried out. 
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In an embodiment an automated specimen transfer mechanism is provided to 
transport cytological specimens between a storage location and an examination 
location. 

In an embodiment of the invention a marking system marks selected areas of 
a cytological specimen at which prescribed characteristics appear, such marking being 
either automatically, semi-automatically or manually initiated. 

In an embodiment of the invention classification of a cytological specimen is 
authorized if an authorized identification is associated with the specimen, and such 
classification may be prevented if such authorized identification is not found. 

In an embodiment of the invention improvements are provided to an automated 
microscope, including, for example, one or more of focus control, light intensity 
control, positioning control, and lens or objective magnification changing. 

These and other objects, advantages and features of the present invention will 
become evident to those of ordinary skill in the art after having read the following 
detailed description of the preferred embodiment 

It is noted here that the published articles cited herein are specifically 
incorporated by reference. 

Moreover, it is noted here that the invention is described herein mainly with 
respect to classification of cytological specimens in the form of a cervical smear, as 
typically is done in connection with a Pap test. However, it will be appreciated that 
this is but one example of the application of the principles of the invention which may 
be used to classify other cytological specimens. 

Pgl EF PKSCRTPTTON OF THE DRAWINfiS 
In the annexed drawings: 

Figure 1A is a schematic illustration of an automated cytological specimen 
classification system according to the invention; 

Figure IB is a schematic illustration of the automatic fine focus adjustment of 
the system of Figure 1 A; 

Figure 2 is a block diagram of an automated cytological specimen screening 
device in accordance with the present invention with particular emphasis on 
classification components; 
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Figures 3A and 3B present a schematic flow chart short-hand representation 
of the method of classifying objects in an exemplary operation of the invention. 

Figure 4 is a block diagram representation of the method of cell classification 
used by the device of Figure 1; 

5 Figure 5 is a block diagram of a classification of a slide having no 

pathological cells; and 

Figure 6 is a block diagram of a classification of a slide having 50 
pathological cells. 

10 PESCBffTION OF THE PREFERRED AND AT.TRR NATR FMBOnTX^rry 

With reference to the drawings in which like reference numerals depict like 
items, and specifically to Figure 1A, there is illustrated a neural network based 
automated cytological specimen classification (sometimes referred to as screening) 
screening device or system 10 in accordance with the present invention. The 
15 screening device 10 includes an automated microscope system 12, a camera 14, a 

barcode reader 16, a slide marker 18, and a computer processing system 20. 

Briefly, the screening device 10 is used to classify cytological specimens, in 
the preferred embodiment to determine and/or to help to determine whether a 
cytological specimen contained on a slide S (or on or in some other support, 
container, etc.) includes characteristics or features of interest. Exemplary 
characteristics are those which are had by malignant or pre- malignant cells in what 
is commonly known as a Pap smear. In an exemplary embodiment described in detail 
below, the automated microscope system 12 makes a low resolution examination of 
the specimen during which some working parameters, such as the location of the 
25 specimen on the slide, focus, and/or mumination level (for optimum viewing in the 

subsequent high resolution examination), are determined. Authorization to conduct 
the exaniination, e.g., using the barcode reader 14 to sense whether the barcode on 
a slide is proper, also may be determined prior to or during the low resolution 
examination. Thereafter, with reliance on such working parameters and 
authorization, a high resolution examination of the specimen is made by the 
microscope system 12. Based on information obtained during such high resolution 
exaniination, primary, secondary and tertiary classification procedures are carried out 
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to determine, for example, whether malignant or pre-malignant cells are contained in 
the specimen. Preferably the primary and secondary techniques are automated and 
the tertiary classification is carried out manually, i.e., by a person. However, 
consistent with the invention other types of primary, secondary and/or tertiary image 
processing techniques may be employed. 

Operation of the system 10 is generally under the control of the computer 
system 20. Accordingly, such computer system includes a general purpose computer 
20a, such as an AT type microcomputer, and an image processor 20b, such as one 
sold under the U.S. registered trademark PIPE by ASPEX INCORPORATED, and 
a neurocomputer 82, such as that sold under the U.S. registered trademark ANZA 
PLUS bh HNC, Incorporated. 

The automated microscope system 12 includes a number of elements designed 
to facilitate the quick and easy handling of specimen slides. One such element is a 
robotic slide handler 22 which, upon appropriate commands from the computer 
system 20, moves the specimen slides from a holder 23, called a cassette, to a 
motorized movable stage 24 for transport into and within the optical path of the 
microscope for cell classification, and then back to the cassette after classification. 
Mounted to (preferably bolted to) the motorized stage 24 is a slide support bridge 
(sometimes referred to as a tooling plate or tooling fixture) 25 upon which the slide 
is held for movement through the irdcroscope system 12. The bridge 25 has one or 
more passages 26 (preferably four) in the area below where the slide is positioned and 
opening to face the slide to allow the creation of a vacuum under the slide to hold it 
firmly down upon and in place on the bridge. The tooling fixture bridge 25 also 
includes a relief area or slot 27 to allow the robotic slide handler 22 to grip the slide 
during placement on and removal from the tooling fixture bridge 25. The slot 27 also 
provides space for fflwninating light from a light source 28 to travel along an optical 
path designated 29 to the microscope objectives 30, 31. 

The motorized stage 24 is mounted upon cross roller bearings 32 and powered 
by two stepping motors 33, 34 with associated drivers by Compumotor together with 
a Parker Compumotor PC23 interface controller 35 to provide precise movement of 
the slide relative to the optical path 29 and viewing area of the microscope. The 
bearings 32 are mounted to the microscope base or frame 40, which in turn is 
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attached by conventional springs and vibration damping shock absorbers 41 to a heavy 
(e.g., 500 pounds) granite base 42. Support of bearings 32 also may be by springs 
and/or shock absorbers 43. 

Hie stage motors 33, 34 are controlled by the motor controller 44 (the 

5 mentioned FC23 interface) which converts commands from the computer processing 

system 20 into appropriate signals to cause the motors 33, 34 to move the stage to the 
prescribed locations. The precise location of the stage 24, and thus the slide, is 
detected by the position encoder 44. Such position encoder 44 may be a conventional 
device that produces pulses representative of movement or location of the stage. 

10 Those pulses, as is conventional, may be decoded by the computer system 20 to 

identify the current location of the stage 24, e.g., relative to a home or reference 
location or relative to some other location. An exemplary position encoder is sold 
by Heidenhain. 

The automated microscope 12 also includes features to provide fast, precise 

15 imaging of each area and/or of selected areas of a slide positioned on the bridge 25, 

as is described further below. The optical system 45 of the automated microscope 
12 includes an objective carriage 45, and an auto focusing mechanism 46. The light 
source 28 includes a lamp 47 of constant color temperature, light intensity control 
filters 50, an automated iris or diaphragm 51, and associated reflectors, prisms, 

20 lenses, light conductors, etc. , schematically represented at 52, to send light along the 

optical path 29 to muminate the specimen from below. Qf desired, fflumination of 
the specimen can be from above.) 

The objective carriage 45 moves the appropriate magnification objective 30 
and 31 (say, respectively, of 5X and 20X iragnification), or lens system, into place 

25 in the optical path 29 to provide for low or high resolution viewing of the slide, 

respectively, as desired during the particular phase of operation. A motor 53 which 
is controlled via a connection 54 by the computer system 20 selectively moves the 
carriage 45 and respective objectives 30, 31 into the optical path 29. limit switches 
55, 56 sense maximum travel of the carriage 45 and cooperate with the computer 

30 system 20 m standard fashion to prevent over-travel of the carriage. A conventional 

adjustment 57 is schematicaUy shown for parcentering, i.e., centering of the 



WO 91/15826 PCT/US91/02138 

9 

respective objectives 30, 31 in optical path 29 regardless of whether the carriage 45 
has moved to position one or the other objectives in the optical path. 

The autoiris 51, such as an adjustable aperture or diaphragm, is for controlling 
the intensity of light transmitted to the slide and in the optical path 29 of the 
microscope 12 to the objectives 30, 31. The autoiris automatically adjusts the 
transmitted light intensity according to which objective 30, 31 the carriage 45 has 
positioned in the optical path 29. A motor 60 controlled by the computer system 20 
adjusts the iris 51 to respective relatively more open and relatively more closed 
conditions at the same time that motor 53 moves the carriage 45 to place respective 
low and high resolution objectives in the optical path 29. 

The filters 50 may be counter rotating variable neutral density linear polarizer 
filters 50a, 50b positioned in the optical path 29 between the light source 41 and the 
iris 51 to provide further control of the light intensity transmitted into the optical path 
without affecting the color temperature of the light. A motor 61 rotates the filters 
50a, 50b under control of computer system 20, which may automatically call for an 
increase or a decrease in slide illumination intensity, e.g., as requested by a user, as 
the light source ages and/or is changed, etc. By rotating the polarizers 50a, 50b, the 
extent that they cross or are parallel and, thus, the amount of transmission 
therethrough can be controlled. 

The automated microscope includes a coarse focus adjustment mechanism and 
a fine focus adjustment mechanism 69 (Fig. IB), both being of conventional design, 
and, therefore, neither of which is shown in detail. The coarse focus adjustment may 
be operated by a micrometer type control that can be operated, e.g. , turned manually. 
The fine focus adjustment also can be operated manually if desired. However, the 
fine focus adjustment preferably is operated automatically by the autofocus 46 in the 
manner described below. (Other types of focus adjustments also may be employed.) 

As is seen in Figure IB, the autofocus 46 includes a light source 70, a bi-cell 
71 (a device with two photocells or other photosensitive devices 72, 73 in a housing 
74 with a mask formed by a pair of openings 75, 76 for guiding light to the 
respective photocells,) a differential amplifier 77 with an offset connection to the 
computer 20, a piezoelectric device 78, and a mechanical coupling 79 to the fine 
focus 69. light from the source 70 is reflected off the top surface of the cover slip 
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S' on slide S. The photosensors 72, 73 produce electrical outputs representing the 
position of the slide relative to the light source and the bi-cell 71. Differential 
amplifier 77 determines the difference between the photosensor outputs (it being 
desirable that such difference be minimized). An offset voltage can be provided via 
5 the computer and a digital to analog converter (not shown) to compensate for the 

thickness of the cover slip S' so that the actual point of focus is at the surface of the 
slide S or a desired depth into the specimen on the slide S. The output from the 
differential amplifier 77 may be further amplified by an amplifier 77a, and the output 
from such amplifier is used to provide a voltage input to the piezoelectric device 78. 

10 The piezoelectric device then mechanically operates the fine focus 69 control of the 

microscope 12 via the mechanical connection 79. 

Another focus control for the microscope 12 also may be provided. Such 
focus control may rely on image processing, as is described further below, to 
determine the degree of focus that an object is seen by the camera 14 in particular. 

15 Such image processing focus control can be used both to make a focus map of the 

slide S and also to control the fine focus adjusting mechanism 69 and/or the coarse 
focus adjusting mechanism to bring the image into focus for the camera. 

The image processing focusing described may be carried out during high and 
low resolution viewing. During low resolution viewing focal information is 

20 correlated with position coordinates from the position encoder 44 to compose a focal 

map, and the resulting focal map is stored in computer memory. During high 
resolution viewing, the computer system 20 provides the focal information from the 
stored focal map in accordance with the location of the viewing field as determined 
by the position encoder 44 to adjust focus by mechanically controlling the coarse 

25 and/or fine focus adjustment mechanisms of the microscope 12. This allows for fast 

focusing during the high resolution scan. 

Alternatively, focusing can be carried out during other portions of the low 
and/or high resolution examinations of the slide. As an example, the focusing function 
can be carried out prior to the initial low resolution examination of the slide and/or 

30 prior to the first high resolution examination of the slide. In the high resolution 

examination, as is described further below, several specific areas or "tiles" of the 
slide are examined, and focusing function can be carried out before each tile is 
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examined or after the initial focusing such focusing can be carried out each time a 
predetermined number, e.g., five, of tiles has been examined. 

The barcode reader 16 may be a conventional device, such as an integral 
barcode reader system, which is sold by Symbol Technologies, Inc of Bohemia, New 
York under the trademark LaserScan 6X20. Such barcode reader 16 is positioned to 
view a selected area of a slide once it has been transported to the stage 24 by the 
robotic slide handler 22. In the preferred embodiment each slide presented to the 
system 10 for classification must contain a barcode. The barcode contains relevant 
information necessary in coordinating the classification results to the slide, e.g., the 
doctor providing the slide and the patient from which the specimen on the slide was 
obtained. The barcode reader 16 reads the encoded information from the slide and 
provides that information to the computer system 20 via connection 80 for storage and 
future correlation with test (U., classification) results. In the event that a slide is 
presented to the screening system 10 without a barcode, or with an improper or 
unreadable barcode, the slide will be rejected and returned without classification to 
the cassette 23 by the robotic slide handler 22. In other embodiments the barcode and 
barcode reader could be replaced by a system performing similar functions, such as 
a set of characters and an optical character reader. 

When a physician collects a cytological specimen from a patient, the physician 
may securely affix a barcode label (e.g., an adhesive label) to the slide on which the 
specimen is placed and may securely affix a corresponding label and/or barcode to 
the patient's chart. The chart may be retained by the physician. When the system 
10 examines a slide, preferably it also prints a report of the results of such 
examination. The barcode information read by the system 10 preferably is printed 
on such report. Then the physician can compare the printed barcode information with 
the corresponding barcode or like information on the patient's chart to confirm 
accurate matching of the report to the patient. The physician also can supply the 
laboratory, which is using the system 10 to classify the cytological specimen, written 
information, such as the patient's name and the physician's name, that can be 
correlated with the barcode and printed automatically on the report concerning the 
classification specimen. 
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The camera 14 is positioned in the microscope's optical path 29 to capture a 
focused, magnified electronic image of the area of the slide being viewed. The 
camera 14 feeds the electronic image to the computer system 20 via a connection 81 
for classification of the cells appearing in the imaged area. The camera may be a 
conventional three chip charge coupled RGB camera, such as one made by Sony, or 
other camera able to provide suitable information of the specimen, i.e., an image, to 
the computer. Such image is preferably represented by electrical signals which can 
be processed by the image processor 20b of the computer system 20. 

In the computer system 20 a number of image processing and evaluation 
functions are performed. These include determining where on a slide is there 
specimen material actually located, whether there is adequate specimen material to 
perform a meaningful classification, and the primary classification which does a low 
level filtration or screening, e.g., based on morphology that can be evaluated 
algorithmicaUy, The neurocomputor portion 82 of the computer system 20 provides 
the secondary classification, doing a higher level filtration or screening based upon 
training of the neurocomputor, e .g., as is disclosed in the parent patent application 
mentioned above and the references mentioned above as well as according to the 
description presented herein. Electronic image representations of cells which are 
classified by the primary and secondary classifiers as being suspect are stored in the 
computer memory, in disk memory, or in some other mass storage device for further 
(tertiary) classification by a person trained to detect the truly abnormal cells. The 
locations of such suspect cells on the slide also are stored in the computer. 
Subsequently, when the tertiary classifier (technician) can review the images of such 
suspect cells, they can be inspected or examined by viewing such stored images on 
25 a video monitor; and the technician can make a final determination as to whether each 

of such suspect cells is truly abnormal, e.g., malignant or pre-malignant or otherwise 
of interest. 

When the tertiary classifier finds such an abnormal cell, she or he may use a 
mouse or some other device coupled to the computer 20a to point to such cell. Such 
cell then is identified or is tagged (e.g., in software or an electronic data 
representation of such cell) for convenient recall and redisplay for viewing and 
examination by a supervisor, pathologist, etc. An image of such cell may be printed 
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onto paper. Also, the physical location of such cell on the slide can be marked, e.g. , 
by placing an ink dot proximate such cell on the slide or on the cover slip thereof. 

More particularly, after the tertiary classifier has identified the truly abnormal 
cells, the computer system 20 commands the motor control 35 and stage motors 33, 
34 to position the respective areas of the slide having an abnormal cell under the slide 
marker 18. (If the slide had been returned to the cassette 23, the robot handler 22 
first replaces the slide back on the bridge in the same position as it had been before 
when inspected by the camera and so on.) The slide marker 18 then is controlled by 
the computer 20 to mate a small dot, approximately .25mm in diameter, on the slide 
in the (several) area(s) of the slide where such abnormal ceil(s) is (are) located by 
operation of a solenoid (not shown) to dab ink via an arm 18a onto the slide. Such 
marking is similar to the manner in which silicon wafers are marked for specified 
purposes in the field of such wafer manufacturing and inspection. The computer 
system 20 will then command the robotic slide handler 22 to return the classified slide 
to the cassette 23. Another slide then can be marked as described until all slides in 
the cassette have been screened. The slide marker may be one of the type sold by 
Xandex. 

Motion of the stage 24 to bring portions of a slide S that is held on the bridge 
25 is under control of the computer 20. The computer sends control signals to the 
interface control circuit 35, which in turn sends appropriate signals to driver circuits 
associated with the stepper motors 33, 34 to move the stage in the X and Y directions 
indicated in Figure 1. Limit switches 90, 91, 92, 93 detect maximum travel of the 
stage and are coupled to the computer 20 to limit travel beyond maximum limits, as 
is conventional. Feedback to indicate the actual location of the stage or the relative 
location compared to a reference location is obtained by the position encoder 44, as 
is conventional. 

The actual travel path of the stage to move the slide to desired locations 
relative to the optical path 29 and/or to move the slide to a position convenient for 
pick up and placement by the robot handler 22 can be programmed into the computer 
using conventional techniques. The moving of the stage 24, bridge 25 and slide S 
relative to the optical path 29 to place specific cells into the optical path for viewing 
during respective low and high resolution examinations by the respective objectives 
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30, 31 and when specific cells are desired to be reexamined also can be controlled 
automatically as a function of location information that can be stored in the computer 
and/or mass storage memory. Further, the locations to move cells for automatic 
dotting by the marker 18 and arm 18a also can be controlled automatically based on 
stored location information that is delivered to the computer 20. Appropriate offsets 
may be added to location information, e.g., to move a cell desired for marking 
relative to the marker arm 18a taking into consideration the location of the marker 
arm 18a, the location of the optical path, and the desire to place the mark adjacent 
the cell (and not directly on the cell so as to obscure the cell in case it is to be viewed 
subsequently). 

Considering, now, the robotic slide handler 22, such device is similar to a 
silicon wafer robotic handler used in the semiconductor industry for automatically 
handling, moving, etc., silicon wafers on which integrated circuits or the like are 
formed. Accordingly, such handler 22 includes an arm 100, which is mounted 
relative to the microscope frame 40 for movement to move slides from the cassette 
23 to the stage 25 and vice versa. Motive means 101 for mounting and moving such 
arm are provided in a support housing 102. Appropriate joints, etc., may be 
provided in the arm 100, depending on the degrees of freedom of movement desired. 
At the end of the arm 100 is a support foot 103 on which a slide actuaUy is carried. 

The foot 103 includes a top surface 104 that can fit beneath a slide S, which 
is shown in the cassette. An opening 105 in the surface 104 of the foot and a passage 
to such opening provides a source of vacuum to hold the slide S to the foot. The 
vacuum opening is coupled via a passage 106 and valve 107 to a vacuum source 110. 
Such vacuum source 110 also is coupled via vacuum line 111 and valve 112 to the 
vacuum lines and openings 26 in the stage 25. 

The cassette 23 is mounted on an elevator 113 and preferably is held thereon 
by a vacuum drawn through openings 114, vacuum line 115 and valve 116. The 
valve 116 preferably is manually controlled (energized or deenergized) by a manually 
operated electrical switch 117. Closure of such switch 117 energizes the valve 116 
to provide a vacuum that holds the cassette onto the elevator 113; opening of the 
switch deenergizes the valve to release the cassette from the elevator enabling easy 
removal therefrom. 
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The elevator 113 includes a lift mechanism 120 and a support platform 121. 
An electronic control circuit 122, which can be controlled by the computer 20, 
provides information to the control circuit 122 to identify from where to obtain the 
next slide (e.g., from the cassette and where from the cassette or from the stage 25) 
and to where to move the slide. The control circuit 122 may be of the conventional 
type used in robot systems that handle silicon wafers, as was mentioned earlier. 
Thus, for example, upon sending the appropriate signal to the control circuit 122, the 
computer can initiate an operational cycle of the handler 22, say to remove a slide 
from the cassette 23 and to place the slide on the stage 25, and vice versa. 

Coordinated operation of the arm 100, foot 103, elevator 113, stage 24, and 
control circuit 122 is controlled by the computer 20. As one example, to pick up a 
slide S from one particular location of many slide storage locations in the cassette 23, 
the elevator 113 lifts the cassette 23 slightly to provide clearance for the foot 103 
beneath the particular slide S. The motive mechanism 101 moves the robot arm 100 
and foot 103 to place the foot beneath the slide. The valve 107 then is energized by 
the control circuit to supply vacuum to the opening 105. Then the elevator 113 
lowers the cassette 23 to lower the slide S onto the top surface 104 of the foot 103. 
The slide then is held onto such surface 104 by the vacuum and by gravity. 

The computer by now has caused the stage 24 to move the bridge 25 to slide 
loading position. The arm 100 then swings to move the slide S into alignment above 
the openings 26 at an appropriate location on the bridge 25. The bridge 25 and/or 
the foot 103 are moved vertically relative to each other so that the slide is placed onto 
the top surface of the bridge 25. The foot 103 fits in a recess 27 in the bridge. The 
valve 112 is energized to supply vacuum to the openings 26 to hold the slide on the 
stage, and the valve 107 is deenergized to release the vacuum holding the slide to the 
foot. 

Similar operation can be used to move the slide from the bridge 25 back to the 
cassette, and so on. 

Referring to Figure 2, the screening device 10 is shown with particular 
emphasis on the classification elements embodied in the computer system 20. The 
computer system 20 includes an image processor and digitizer 20b, a neurocomputer 
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82, an output monitor 154, and a general processor 20a with peripherals for printing, 
storage, etc. 

The general processor 20a is preferably an IBM PC/AT or compatible 
although it may be another computer-type device suitable for efficient execution of 
5 the functions described herein. The processor 20a controls the functioning and the 

flow of data between components of the device 10, causes execution of additional 
primary feature extraction algorithms such as an integrated optical density function 
and handles the storage of image and classification information. The processor 20a 
additionally controls peripheral devices such as a printer 158, floppy and hard disk 

10 storage devices 160, 162, and the barcode reader 16, slide marker 18, autofocus 46, 

robotic slide handler 22, stage motor controller 35, and objective carriage 45 
components described more fully above. 

The image processor and digitizer 20b also performs primary cell classification 
functions such as thresholding and erosion and dilation. In the preferred embodiment, 

15 the image processor and digitizer 20b is a commercially available low level 

morphological feature extraction image classifier such as the ASPEX Incorporated 
PIPE (TM) image processor which includes an image digitization function. The PIPE 
(TM) image processor is described more fully in U. S. Patent No. 4,601,055, the 
entire disclosure of which hereby is incorporated by reference. Alternatively, the 

20 image processing and digitization function could be separated into two or more 

components. 

Secondary group processing cell classification is performed by the 
neurocomputer 82. The neurocomputer 82 is a computer embodiment of a neural 
network trained to identify suspect cells. In this embodiment the parallel structure 

25 of a three-layer backpropagation neural network is emulated with pipelined serial 

processing techniques executed on one of a host of commercially available 
neurocomputer accelerator boards. The operation of these neurocomputers is 
discussed in the Spectrum reference cited. The neural network is preferably 
implemented on an Anza Plus processor, which is a commercially available 

30 neurocomputer of Science Hecht- Nielsen Neurocomputers (HNC) (see the 

Hecht-Nielsen reference above). Alternatively, secondary cell classification functions 
could be performed using a template matching algorithm designed to identify shapes 
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known to be typical of a pathological cell. A template matching or other group 
processing algorithm could be efficiently implemented in a parallel distributed 
processing network, for example. Another alternative secondary classification 
embodiment is a holographic image processor designed to perform group based 
classification. 

Referring to Figures 3A, 3B and 3C a flow diagram of the operation of the 
cell classification method of the present invention is outlined. Initialization of the 
system 10 is carried out at block 199. During initialization the image processor 20b 
is initialized so as to be ready to receive the first electronic information representing 
the image received by the camera 14 and to process the image information. Also the 
stage 24 is initialized to place it in a reference location, sometimes referred to as a 
home location so that the computer 20 can expect to know that the stage is at such 
location and can determine future locations based on the home location. The robot 
handler 22 is initialized, too, to place the various portions thereof in a home position 
to so that future locations and movements thereof can be determined from the initial 
home position. The neural net also is initialized in conventional fashion. 

To begin operation, the robotic slide handler 22 grabs the first slide from the 
cassette 23 and transports it to the motorized stage 24 (block 200). If the robotic 
slide handler was unable to find the next slide, such as when all slides in the cassette 
23 have been classified, a message is conveyed to the operator (205). Assuming a 
slide is present, the barcode reader 16 reads the barcode information from the slide 
(210) and passes the information to general processor 20a (215) for correlation with 
future classification data. Cells not having barcodes or barcodes that are not readable 
are rejected and the next slide is processed (216). Next, the general processor 20a 
commands the stage motor controller 35 and motors 33, 34 to move the stage 24 and 
slide into the optical path 29 of the microscope system 12 for the low resolution pass 
(220). 

In the low resolution pass, the carriage 45 will move the low resolution 
objective 30 into the optical path 29, and the autoiris 51 will automatically adjust the 
lighting for the low resolution objective (225). A relatively quick scan of the slide 
is then made to find the areas of the slide having cellular matter (230). If the no 
areas having cellular matter or an adequate amount of matter for valid classification 
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is found on the slide, then the slide is identified as containing insufficient cellular 
matter to perform a meaningful test (235); the slide then may be rejected (240) and 
screening of the next slide may begin (200). 

To determine whether there is adequate cellular material on the slide or at 
various locations on the slide and also to determine the focus map for the slide, the 
following may be carried out by the computer system 20. First, a scan route is 
determined so mat a plurality of areas on the slide can be viewed sequentially. Such 
areas may be located in a straight line along the length of the slide S or in some other 
arrangement on the slide. As an example, a plurality of areas sequentially located 
along a serpentine path along the slide are viewed. Each such area will be designated 
hereinafter a "macro-tile". 

When a particular macro-tile is located in the optical path 29, the camera 14 
takes a picture thereof. The macro-tile may be, for example, 2mm by 2mm in size. 
In the image processor 20b, using an ISMAP program or algorithm available from 
ASPEX Incorporated, New York, New York, (iconic to symbolic mapper) the 
macro-tile is subdivided into sixteen areas referred to below as "tiles" and a sharpness 
image and a gray scale image are made. These images are used to determine whether 
there is and/or how much there is of cellular material in the macro-tile. A histogram 
of the absolute value of the difference between such images may be used in the focus 
20 function of the computer 20 to determine a focus map for the slide. 

More particularly, the value of the optical transmission characteristics of the 
macro-tile is made; and simultaneously two successive 3 by 3 Gausian filtrations are 
made to provide a 5 by 5 Gausian result. The difference between the two is taken 
and is converted to absolute value, which represents a sharpness image that can be 
25 used for the focus map. 

In other words, a sharpness image is obtained for the macro- tile; and a 
synthetically created gray scale map that is the size of the macro-tile is made. The 
gray scale map has multiple areas that correspond to the respective tiles of the 
macro-tile. The incremental gray scale is determined according to the values in the 
30 sharpness image using histogram techniques. 

If any of the bins in the histogram (which represent respective tiles in the 
macro-tile) is above a threshold value, then the particular tile is noted for high 
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resolution examination and processing because there appears to be adequate cellular 
material there. Furthermore, by taking a further histogram of the absolute value of 
the difference between the original transition characteristic of the macro-tile and the 
gray scale Gaussian filtered image, one can determine the particular maximum for the 
tile; and such maximum may be used as a representation of the optimum focus 
condition for the automated microscope for viewing the particular tile during the high 
resolution pass. See block (250) in Fig. 3. 

The advantage of detennining whether there is adequate sample on the slide 
for classification during the low resolution pass in the microscope 12 is that such 
determination can be made relatively quickly compared with the time needed to make 
the same determination using the high resolution objective 31. The advantage of 
deterniining, during the low resolution pass, which tiles will require further 
examination in the high resolution pass, is to save the time needed unnecessarily to 
examine tiles that do not have cellular material or adequate cellular material there. 

The scan pattern of the areas (tiles) of interest is made (255) and focal 
information for each tile is correlated with position coordinate information for that 
tile from the position encoder 44 to provide a focus map (260). 

All information necessary to perform a high resolution scan of the slide is now 
available. To commence the high resolution pass in the microscope 12, the computer 
20 operates the motor 53 to move the carriage 45 to place the high resolution 
objective 31 into the microscope's optical path 29 (265). The lighting is 
automatically adjusted by the autoiris 52 and motor 60 for the high resolution 
objective, and the stage 24 is moved to bring the first tile or segment of the slide S 
intended to be examined into the viewing field of the objective 31. The computer 20a 
commands the autofocus 46 to adjust focus for the area, segment or tile (270) of the 
specimen under examination by providing ah appropriate offset signal on line 79 to 
the differential amplifier 77. 

The camera 14 obtains a color video picture of the focused image (275, Fig. 
3B), and that image is digitized and is stored (sometimes referred to as frame 
grabbing), as is described further herein. The stage 24 then moves the next tile or 
segment into view and appropriate focal adjustments are made in accordance with the 
focus map (280). If the last segment (tile) of the slide has been reached (285), a flag 
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is set and processing of the slide will be discontinued after screening of that segment 
(290). The stage 24 may wait at a location until an image of a tile is obtained. 
Preferably, though, such waiting time should be minimized to ininimize the time 
needed to examine a slide. 
5 Preferably image processing of one or more than one segment or tile can be 

carried out in the image processor 20b simultaneously while an image of another 
segment is being obtained by the camera 14. 

The color components of the video representation of a segment are summed 
to provide a monochrome image (295), and that image is passed to the image 
10 processor and digitizer 20b where the primary classification of the segment begins. 

Initially, the image processor and digitizer 20b performs an adaptive threshold 
operation on the video image to enhance the image contrast and eliminate noise from 
the background (300). This thresholded image is then down sampled to a manageable 
digital representation (305). The image processor and digitizer 20b can then perform 
15 erosion and non-connecting dilation operations on the digital image to separate the 

objects in the segment (310. The monochrome and resultant filtered images are 
transferred to RAM of the general processor 20a (315, 320). 

The erosion and dilation techniques are conventional image processing 
techniques. They eliminate the effect of overlapping cells in which dark areas on 
may appear due to the increased density of the overlap rather than due to enlarged or 
especially dark cell nuclei. It is usually the darkened nuclei or large size nuclei that 
are detected during the integrated optical density (IOD) evaluation made in the low 
level classification procedure described further below. The erosion and dilation 
technique also enhances the accurate examination of the cells during the low level 
25 classification. 

An object count also may be performed at this time to find out how many 
objects have passed the erosion and dilation. That number of objects is approximately 
representative of the number of objects in the specimen. It is desirable that at least 
a minimum number of objects be included in the specimen, for if the sample size is 
too small, then the test may not produce meaningfully accurate or reliable test results. 
The object count or other means to determine the "validity" of the sample may be 
taken at other times in the described process of the invention. As is described herein, 
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it is desirable according to the preferred embodiment that a person be the one making 
the tertiary classification; that person ordinarily is expected to be reviewing images 
of several cells which the primary and secondary classifications had determined to 
have a relatively high probability of being malignant or pre-malignant. It is desirable 
that the person know that if the number of cells being reviewed manually is zero or 
is relatively low, that is due to the fact that the other cells are healthy and not due to 
the fact that there were not enough cells to examine in the specimen. 

The processor 20a then performs further primary feature extraction 
classification on the segment such as with an integrated optical density (IOD) 
algorithm (325). Other morphological algorithms may alternatively or additionally 
be used to classify based on features, such as color or features relating to DNA ploidy 
analysis, immunohistochemistry, DNA hybridization, etc. These feature extraction 
algorithms isolate certain objects which possess features typically known to be present 
in pathologic cells, such as a dark cell nucleus which is abnormally large in relation 
to the rest of the cell. The centroids of objects identified by primary classification 
as being possibly pathological are catalogued and stored in RAM of the processor 20a 
(330). If no objects have been identified (335, Fig. 3C), classification begins on the 
next segment (295). 

Identified objects, i.e., electronic image representations of those which have 
not been eliminated by the low level classification, are transferred individually to the 
neurocomputer 82 as digital areas around the object centroid (340). The 
neurocomputer 82 will perform secondary classification on the objects in accordance 
with its previously completed training which is described more fully below (345). 
Additionally, an object count may be made at this point (and/or possibly elsewhere 
in the process, as is mentioned above) to determine if the neurocomputer 82 has 
received a sufficient number of cells from the primary classifier to indicate a valid 
test. For objects identified by the neurocomputer 82 as suspect (350), the color 
representation of a suitable area surrounding the centroid is retrieved from disk and 
transferred to the high resolution display board of the high resolution monitor (355). 
Cells having a classification less than the threshold level are discarded and the next 
centroid is obtained from the general processor (340) and classified (345). When it 
is determined that all locations, tiles, on the display board are occupied (360), the 
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total image is transferred to the general computer 20a for temporary storage (365) 
until all cells on the slide have been screened, and the high resolution board is cleared 
(370) • Once all centroids found in the primary classifier for mat segment have been 
classified by the neurocomputer (375), the image for another segment is grabbed 
5 (275), and classification for that segment is performed (295-375). 

When all segments on a slide have been screened (380), the high resolution 
color images for the slide are transferred to disk (385) for storage until a convenient 
time for display on the high resolution monitor and tertiary classification by a 
cytotechnician or cytologist All arrays of suspect cells may be tagged in memory 
10 with information obtained from the barcode to identify the slide on which they were 

found. The location of the suspect cells on the slide also may be physically marked 
on the slide by the slide marker 18. A report indicating the test results for that slide, 
which is correlated with the barcode information obtained from that slide, may be 
printed on the printer 158 now or later. 

15 The slide is then returned to the cassette 23, and another slide is selected (200) 

to begin the classification process anew. 

It should be recognized that while the image processor and digitizer 20b, the 

general processor 20a, and the neurocomputer 82 are described and shown in Figures 

3a and 3b operating in a serial manner, in actual practice as many functions will be 
20 performed in parallel as is possible. Consequently, the components 20b, 82, 20a may 

process different slide segments or different areas of a segment concurrently, greatly 

reducing the time required to screen a slide. 

Turning to a more in-depth review of the classification method and with 

reference to Figure 4, a block representation of the classification functions of the 
25 screening device 10 is illustrated. Primary classification, such as low level 

morphological feature extraction, is performed as indicated above by both the image 

processor and digitizer 20b and general processor 20a and is represented in Figure 

4 conjunctively as block 400. 

Initially, as described more fully above, the video camera 14 obtains an image 
30 of the cytological specimen which is digitized for classification use. The primary 

classifier 400 first performs an erosion of the image. This erosion operation 

successively peels off layers of pixels from each object in the image so that all of the 
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objects which are smaller in size than the smallest known pathological cell nucleus 
are removed from the image. The remaining objects are then dilated, i.e., regrown, 
by successively adding back layers of pixels to these objects, but they are not dilated 
to the point where they are touching each other. The basic operations of erosion and 
dilation can be found in several sources in the prior art (e.g., Sena, J., "Image 
Analysis and Mathematical Morphology", Academic Press, 1982). 

Based on experience with an engineering prototype, it has been found that for 
every 1,000 objects found in a typical benign Pap smear no more than about 15 
objects will pass the erosion/dilation screen. These relatively few remaining objects 
are then subjected by the primary classifier 400 to an integrated optical density (IOD) 
screen. 

Integrated optical density is the sum of the pixel grey values for each object. 
Pre-malignant cells tend to possess large dark nuclei. The IOD threshold is, 
therefore, set to filter out any object which passes the erosion/dilation screen but 
which has an IOD which is above or below the threshold displayed by a truly pre- 
malignant or malignant cell. For the IS objects which passed the erosion/dilation 
screen, experience shows that no more than ten objects will pass the IOD filter. 
Thus, the average combined filtration of erosion/dilation and IOD reduces an input 
image of 1,000 objects to an output image of ten objects. These ten objects may 
include not only pre-malignant and malignant cells but also other objects with a high 
integrated optical density such as cell clumps, debris, clumps of leucocytes and 
mucus. The task of the secondary classifier 420 is to distinguish the pre-malignant 
and malignant cells from these other objects. 

For the engineering prototype, classifier 420 was a backpropagation neural 
network hosted on an Anza Plus neurocomputer coprocessor board resident on an 
IBM PC. The backpropagation network was trained with a training set of several 
hundred known benign and pre-malignant or malignant cells to associate a benign 
image with a diagnosis of 0.1 and a non-benign image with a diagnosis of 0.9. 

In actual operation, the secondary classifier 420 is presented with images 
passed to it by primary classifier 400. These are images which may be similar but 
are not identical to those used to train classifier 420. The task for secondary 
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classifier 420 is thus to generalize from its training process to successfully classify 
benign and non-benign images to which it was not previously exposed. 

One major advantage of the present invention over the prior art resides in the 
fact that each image presented to the secondary classifier 420 is pre-centered by the 
primary classifier 400 on the centroid of the suspect cell nucleus. This is 
accomplished because erosion/dilation and IOD based filtration automatically results 
in a centering of each suspect image around its dark centroid. In prior art attempts 
to utilize neural networks and other high-level template matching pattern classifiers 
for image recognition, difficulty has been encountered in consistently presenting the 
classifier with the centroid of the image requiring classification. To use an example 
from another application domain, backpropagation networks are excellent at reading 
handwritten zip code digits but have difficulty in finding where the zip code is on the 
envelope. The present invention overcomes this difficulty in the domain of cytology. 

In an engineering prototype, a 128 x 128 pixel image was stored around each 
centroid which passed the low level filters of classifier 400. A 64 x 64 window, also 
centered around the same centroid, was then compressed using pixel averaging to a 
24 x 24 pixel image. Note that this image is still centered on the same large, dark, 
image which passed the erosion/dilation and IOD filters of classifier 400. A set of 
several hundred of these pre-centered 24 x 24 pixel images of known benign and 
non-benign cells was used to train classifier 420. During feed-forward, Le., post- 
training operation, when classifier 420 is presented with new images it did not 
encounter during its training it must generalize from the training set images to select 
the diagnostic category which most closely matches the new image. A fundamental 
advantage of the present invention over the prior art is that during this feed-forward, 
post-training phase of its operation, classifier 320 is presented with precisely the same 
type of 24 x 24 pixel images on which it was trained. These images are also centered 
on the centroid of the suspect nucleus by classifier 400 in a manner identical to that 
used to prepare the training set images. This makes the generalization task of 
classifier 420 for easier and thus much more successful than anything found in the 
prior art 

As noted above, classifier 420 is trained to associate a known benign image 
with an output of 0.1 and a known pathological image with an output of 0.9. Such 
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outputs represent, for example, the degree of certainty that a cell is normal or 
abnormal, respectively. When classifier 420 is then presented with new, unknown 
cells, it generalizes from its training and attaches an output to the image. The closer 
that classifier 420 is able to categorize the unknown image into the benign category, 
the closer is its feed-forward output equal to 0. 1. Conversely, the more closely that 
an unknown image appears to classifier 420 to resemble the non-benign images of its 
training set, the closer is its feed forward output for that image equal to 0.9. 

During testing of an engineering prototype in which a backpropagation neural 
network was used for classifier 420, it was found that no truly pre-malignant or 
malignant cell was attached to an output of less than 0.75. In order to provide a 
margin of safety, classifier 420 only screens out images with an output of 0.65 or 
less. Any image which is attached to an output greater than 0.65 is assumed to be 
a suspect pre-malignant or malignant cell. For each of these suspect images, the 
associated 128 x 128 pixel image centered around its centroid is retrieved from image 
memory and displayed as one of a field of 64 such images on a high resolution output 
monitor for final classification by a cytotechnologist. 

All images which are classified by classifier 420 to have an output of 0.65 or 
less are assumed to be benign and are not displayed on the output monitor. During 
testing of the above- described engineering prototype, it was found that classifier 420 
consistently filtered out over 80% of the benign images sent to it by the output of 
classifier 400. In other words, over 80% of the truly benign images which pass the 
erosion/dilation and IOD screens of classifier 400 are assigned an output of less than 

0. 65 by classifier 420, leaving 20% of these images representing suspect benign or 
pre-malignant and malignant cells to be finally classified by the tertiary classifier 440, 

1. e., the cytotechnologist. 

As mentioned above, in an alternate embodiment the classifier 420 may be a 
high level template matching or holographic imaging filter. It is possible to use these 
filters in an efficient overall processing scheme because the object of interest has 
already been identified by the low level feature extraction filter, classifier 400. 

The overall operation of the cell classification system can be summarized with 
reference to Figures 5 and 6. Figure 5 shows the screening of a typical Pap smear 
which contains approximately 100,000 benign cells and other objects. Through 
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erosion/dilation and IOD filters, the primary classifier 400 will filter out 99% of these 
objects, passing approximately 1,000 objects to the secondary classifier 420. 
Classifier 420, which in the tested and preferred embodiment employs a three-layer 
backpropagation neural network, in turn filters out 80% of these 1,000 objects, 
5 passing the images of approximately 200 residual objects deemed to be most suspect 

of pathology to the output monitor for tertiary human inspection 440. These 200 
objects are assembled as two to three fields of 64 objects each. Each object is 
presented as a 128 x 128 pixel image taken from the video input to classifier 400 and 
centered around the suspect cell nucleus. The tertiary classifier 440, the 
10 cytotechnician or cytologist, will then further screen the 200 objects to zero, since all 

were benign. 

The screening of a Pap smear having 50 pathological cells plus the 
approximately 100,000 cells and other objects (classified above) is shown in Figure 
6. The primary classifier 400 will screen the slide down to 1050 cells (50 
pathological cells and 1000 possibly pathological cells). The secondary classifier 420 
will further screen these cells to 250 cells (50 pathologic plus the 200 most suspect 
benign cells). These 250 cells will then be screened by the tertiary human classifier 
440 resulting in 50 cells being classified as pathological. 

The overall result is that instead of examining 100,000 cells under the 
microscope, the cytotechnologist examines 200 to 250 cells presented on a high 
resolution color video screen, each screen containing 64 images, with the attention 
of the cytotechnologist focused on inspection of the center of each of the 64 128 x 
128 pixel images. 

In the preferred embodiment the invention is used to display images 
representing the first 64 cells in the examined specimen which are most likely to be 
maUgnant or premalignant, i.e., they have characteristics, features, etc., of known 
malignant or premalignant cells. The actual number of images of cells, or number 
of cells themselves, may be more or less than the preferred number of 64. 

Moreover, according to the invention such images may be presented to the 
cytotechnician one at a time, in an array of four, sixteen, or more or less images, and 
those images may be presented at various levels of magnification, which further 
facilitates and enhances accuracy of the tertiary classification. 
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The image or images being screened may also be presented to the 
cytotechnician adjacent exemplary images which have features distinctive of the 
conditions for which the screening is being performed. For example, if a slide is 
being screened for pre-rnalignant or malignant cells, one or more stored images 
having features common to known types of pathological cells, such as a large, dark 
nucleus, may be recalled from memory and displayed adjacent the suspect image. 
Consequently, as the cytotechnician screens cell images which have been classified 
by the primary and secondary classifiers as suspect, he or she will be able to perform 
a side-by-side comparison of suspect cell images with a known pre- malignant and/or 
malignant cell image. This provides a convenient visual reference from which the 
cytotechnician can consistently base his or her tertiary screening criteria. However, 
it is equally useful to employ the side-by-side comparison feature when the human 
classifier performs the initial or sole screening of the slide. 

The false negative rate of cytological screening is known to be a function of 
the ratio of non-pathological to pathological objects which are visually inspected on 
a daily, continuous basis. The present invention drastically reduces the number of 
non- pathological objects which require inspection from 100,000 to 200- 250. In 
addition, all suspect cell nuclei are pointed out to the examiner by their position in 
the center of a 128 x 128 pixel rectangle. The result is a very much less fatigued 
cytotechnologist and a very significant reduction in the false negative rate. 

It is axiomatic of cytological screening that the detection of only one 
premalignant or malignant cell in a specimen is sufficient to warrant a physician's 
further attention to the patient from which the specimen was taken. The converse, 
however, is not always true. The presence of only benign cells in a specimen does 
not conclusively mean that a patient does not have or is not developing the precursors 
of cervical cancer. For example, a pap smear may not have been performed 
properly, and as a result the specimen may contain insufficient cellular matter for a 
reliable test or may be void of the type of cellular matter in which cervical cancer 
most often develops. Whether there is sufficient cellular matter to comprise a sample 
size large enough to possibly constitute a reliable test specimen may be determined, 
as described above, by an object count. A further method of determining the 
reliability of the test, which may or may not be used in conjunction with the object 
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count, is by ascertaining the presence of certain cells or types of cells in the 
specimen. 

Hie lining of the uterus contains columnar shape endometrial cells, and the 
vagina is lined with flat sheet-like cells, known as squamous cells. The 
interconnecting organ, or cervix, includes both these cell types and has a transitional 
region called the squamo-columnar junction where these two cell types meet The 
actual location of the junction within the cervix may move as a woman ages, becomes 
pregnant, etc., and varies among women. It is in this transitional zone between 
squamous and columnar cells that cervical cancer generally develops first. Since the 
malignant or premaUgnant cells developing within the transitional zone may not 
extend down into the vagina until cervical cancer has reached its later stages, it is 
critical that the pap smear swab contact not simply the vagina and lower areas of the 
cervix, but that it contact the transitional zone further within the cervix. The area of 
the cervix above the critical transitional zone, that area most proximate the uterus, 
is lined with columnar cells, known in the cervix as endocervical cells. 
Consequently, the presence of endocervical cells in the specimen indicates that the 
transitional zone, or squamo-columnar junction was, in fact, sampled. 

Since even benign endocervical cells have many features that a premaUgnant 
or malignant cell possesses, the secondary classifier will give these cells a ranking 
relatively higher than most other benign cells. Accordingly, for a specimen having 
no malignant or premaUgnant cells, the endocervical cells will be among the few 
highest ranked cells, and most likely among the highest 64 ranked cells. For this 
reason it is desirable that the classification device display the 64 highest ranked cells 
Ci.e., those cells determined by the invention to be the 64 most likely to be malignant 
or premaUgnant) or other objects for inspection by a cytotechnician, even when those 
cells are all ranked below that of a true premaUgnant or malignant ceU. Attained 
cytotechnician, can easily distinguish an endocervical cell from other benign or 
maUgnant cells, and through the detection of at least one endocervical cell among the 
displayed cells it will be readily apparent that the pap smear swab most likely 
contacted the transitional zone of the cervix. Consequently, a second method of 
*5termiimigraereliabffl^ Further, in the absence of any 

premaUgnant cells or maUgnant cells, since an endocervical ceU wiU be displayed 
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among the first 64 objects displayed if it is present in the specimen, the cytotechnician 
can identify the test specimen as adequate and the test as reliable in very little time. 

Another advantage had by displaying the 64 highest ranked cells is that often 
infections may be ascertained. Since cells that are irritated by many infections, such 
as venereal diseases, etc., also display many of the characteristics of a premalignant 
or malignant cell, the secondary classifier will rank them relatively higher than 
benign, non-irritate cells, while lower than truly premalignant or malignant cells. 
Therefore, for a specimen having no premalignant or malignant cells, the infection 
disturbed cells will generally be displayed among the first screen of highest ranked 
cells for review by cytotechnician. A skilled cytotechnician can then easily detect 
these cells among the relatively few (64) displayed cells, analyze them closer, such 
as by magnifying the image or comparing the infected cell to stored representations 
of infected cells, if desired, and note the specimen as containing evidence of 
infection. 

Consequently, it is apparent that even though the classification device of the 
present invention may rank the specimen cells such that it usually may be determined 
from the first few cells displayed whether the specimen contains pathological cells, 
it may be advantageous that a cytotechnician examine all of the cells of the first 
displayed screen of highest ranked cells to determine the reliability of the test and 
whether there may be indications of certain infections in the specimen. 

As will be evident to those of ordinary skill in the art, the present invention 
overcomes all of the obstacles to practical automated cytological analysis inherent in 
the prior art. This is achieved by the present invention's unique combination of 
feature extraction and template matching techniques. This combination of techniques 
overcomes the requirement for image segmentation found in the prior art. In 
addition, this combination of techniques overcomes the major obstacle of neural 
networks and other computationally intensive template matching techniques for the 
analysis of complex images such as cervical smears. This is achieved by the fact that 
the primary classifiers of the present invention result in an automatic centering of the 
suspect cell nucleus in the input array of the neural network. 

By simultaneously overcoming the requirement for image segmentation and 
also enabling the first practical utilization of neural networks for cytology, the present 
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invention has resulted in the first practical automated cytological screening system. 
Hie present invention has been demonstrated by an engineering prototype to 
successfully analyze standard Pap smears with overlapping and partially obscured 
cells. It has also been shown to perform this analysis within the time period typically 
consumed by completely manual examination. Thus, the unique combination of 
techniques embodied in the present invention has achieved the goal of over twenty 
years of prior art attempts at automated cytological classification. 

However, the invention is not limited to use as a primary screening device, 
but may also be used to rescreen cells which a cytotechnician has already inspected 
and classified as benign. As such, the device provides an effective check to focus the 
cytotechnician's attention to cells which may have been overlooked during the initial 
screening. 

The invention can be designed and trained to classify other objects not 
expected to be found in a typical Pap smear. Such an object may be representative, 
for example, of a herpes virus. 

The invention may also be used in classifying unexpected cells on a slide 
which has previously been screened by a cytologic or cytotechnician on another 
basis. One such instance is in the classification of a Pap smear taken for a 
postmenopausal patient. Such a smear should not contain endometrial cells. Other 
instances are screening for organisms, etc. Consequently, the slide will be analyzed 
by a cytologic for the presence of endometrial cells or other organisms and then 
screened by the invention for other unexpected cells or viruses, such as 
malignant/pre-malignant cells or the herpes virus. 

Another important feature of the present invention is the ability to provide 
adaptive learning for the neural computer 82. Such adaptive technique enables the 
neural computer 82 to be retrained or to be trained with additional information 
representing additional or better defined cells of specified characteristics. 
Characteristics of such cells may be those of malignant or pre- malignant cells; may 
be those of benign cells; may be those of other types of cells or characteristics of 
other types of matter expected to be found in samples intended for examination and 
classification. Further, such adaptive training can be carried out in a laboratory or 
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research facility and the results of such training can be delivered to apparatus 
employing the invention in the field. 

It will also be appreciated that while the invention is described with primary 
reference to Pap smears, the invention is applicable to most any type of cytological 
specimen such as those containing exfoliative or aspirated cells, for example. 
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CLAIMS 

What is claimed is: 

1. A method of deterniining the reliability of cytological screening test, 
comprising the steps of: 

5 a) viewing at least part of a cytological specimen; 

b) creating an image of such view; 

c) producing a digital representation of such image; 

d) ranking objects in such digital representation based on the 
degree to which each respective object has characteristics more likely found in a 

10 typical premalignant or malignant cell than in a typical benign cell; 

e) examining at least some of such objects ranked as having the 
most characteristics more likely found in premalignant or malignant cell than in a 
benign cell for the presence of a predeterined cell type; wherein the presence of at 
least one cell of such predetermined cell type tends to indicate that the test is reliable. 

15 2 - 1116 method of claim 1, wherein said step of ranking includes 

identifying objects in such digital representation as likely to be premalignant or 
malignant based on a detectable feature. 

3. The method of claim 2, wherein said step of ranking further includes 
using a neural computer apparatus to further classify such identified objects as likely 

20 to be malignant or premalignant as a function of the training thereof. 

4. The method of claim 1, further including the step of displaying such 
portion of such objects in man-readable form. 

5. The method of claim 1, wherein such predetermined cell type is an 
endocervical cell. 

25 6. The method of claim 4, wherein said step of displaying includes 

arranging each object in such portion of such objects in the order of its ranking. 

7. Themethod of claim 6, wherein said step of arranging includes 
arranging such objects in a matrix format. 

8. The method of claim 7, wherein said matrix format is an 8 by 8 

30 matrix. 

9. A method of fecUitating the determination of the presence of an 
infection in a cytological specimen, comprising the steps of: 
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a) viewing at least part of a cytological specimen; 

b) creating an image of such view; 

c) producing a digital representation of such image; 

d) ranking objects in such digital representation based on the 
degree to which each respective object has characteristics more likely found in a 
typical premalignant or malignant cell than in a typical benign cell; 

e) examining at least some of such objects ranked as having the 
most characteristics more likely found in premalignant or malignant cell than in a 
benign cell for the presence of cells having indications of infection. 

10. The method of claim 9, wherein said step of ranking includes 
identifying objects in such digital representation as likely to be premalignant or 
malignant based on a detectable feature. 

11. The method of claim 10, wherein said step of ranking further includes 
using a neural computer appartus to further classify such identified objects as likely 
to be malignant or premalignant as a function of the training thereof. 

12. The method of claim 9, including the step of displaying such portion 
of such objects in man-readable form. 

13. The method of claim 12, wherein said step of displaying includes 
arranging each object in such portion of such objects in the order of its ranking. 

14. The method of claim 13, wherein said step of arranging includes 
arranging such objects in a matrix format. 

15. The method of claim 14, wherein said matrix format is an 8 by 8 

matrix. 

Please add the following claims: 

16. A method of classifying a cytological specimen or the like, comprising 
classifying a cytological specimen containing a plurality of objects to 

determine according to a ranking system those objects most likely to be representative 
of a characteristic, 

selecting a number of those objects according to such ranking, and 

displaying images of such selected objects. 

17. The method of claim 16, wherein such characteristic is that 
representative of a malignant or premalignant cell. 
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18. The method of claim 16, wherein said selecting comprises selecting 
those objects which are ranked most likely to be a malignant or premalignant cell 
event though one or more of such objects is not a malignant or premalignant cell. 

19. The method of claim 16, wherein such characteristic includes the 
appearance of a malignant or premalignant cell. 

20. The method of claim 19, wherein such appearance is visual appearance. 

21. The method of claim 20, further including the step of visually 
inspecting said displayed objects. 

22. A method of classifying a cytological specimen, comprising 
performing a classification of a cytological specimen to determine those 

objects in the specimen which are most likely to have the characteristics of 
malignancy or premalignancy, 

said performing a classification including ranking objects in such specimen 
according to such characteristics whereby those objects most likely to be malignant 
or premalignant are ranked closer to one end of the ranking spectrum and those 
objects less likely to be malignant or premalignant are ranked closer to the other end 
of the ranking spectrum, and 

selecting a number of objects for further classification as a function of such 
ranking. 

20 23 ' method of claim 22, further comprising displaying images of such 

selected objects to facilitate further classification by a person. 

24. A method of classifying a cytological specimen to determine whether 
malignant or premalignant cells are included in such specimen, comprising 
perforating a visual inspection of the cytological specimen to determine the presence 
of a malignant or premalignant cell, and rescreening such specimen by performing 
the method of claim 22. 

25. Apparatus for classifying a cytological specimen, comprising 
classifier means for classifying a cytological specimen, 
said classifier means including means for ranking objects in such specimen 

according to the likelihood of having a prescribed characteristic or condition, 



15 



25 



30 



WO 91/15826 PCT/US91/02138 

35 

means for selecting according to rank at least a minimum number of a 
plurality of such objects most likely to have such prescribed characteristic or 
condition, and 

means for facilitating further classification of such selected objects. 

26. The method of claim 25, wherein said means for facilitating further 
classification comprises means for displaying such selected objects for classification 
by visual inspection. 

27. The method of claim 25, wherein said classifier means includes means 
for ranking such objects according to characteristics indicative of a malignant or 
premalignant cell. 

28. TTie method of claim 25, wherein cells having condition requiring 
medical consideration may be included in such specimen, and wherein said classifier 
means includes means for including in such ranking such cells having such condition, 
whereby such cells having such condition may be included as selected objects. 
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